Skip to content

Add mixed-upgrade HP adoption trajectory pipeline (skeleton)#376

Open
sherryzuo wants to merge 19 commits intomainfrom
sz/mixed-upgrade-adoption-trajectory
Open

Add mixed-upgrade HP adoption trajectory pipeline (skeleton)#376
sherryzuo wants to merge 19 commits intomainfrom
sz/mixed-upgrade-adoption-trajectory

Conversation

@sherryzuo
Copy link
Collaborator

Skeleton PR for the mixed-upgrade adoption trajectory work tracked in #375.

Adds the file structure and Justfile recipes; no functional implementation yet. The scripts raise NotImplementedError and the tests are all skipped — subsequent commits will fill them in.

What's included:

  • utils/pre/materialize_mixed_upgrade.py: CLI stub that will assign buildings to upgrades per year and write per-year metadata + load symlinks
  • utils/pre/generate_adoption_scenario_yamls.py: CLI stub that will generate year-indexed scenario YAML entries from a base scenario
  • rate_design/hp_rates/ny/config/adoption/nyca_electrification.yaml: NYISO Gold Book 2025 adoption trajectory config (placeholder fractions to be updated)
  • tests/pre/test_materialize_mixed_upgrade.py: skipped test stubs covering assignment logic, metadata combination, symlink creation, and validation error paths
  • Justfile adoption recipes: materialize-adoption, generate-adoption-scenarios, run-adoption-scenario, run-adoption-all

Closes #375

Skeleton only — no functional implementation yet.  Adds:
- utils/pre/materialize_mixed_upgrade.py: CLI stub that will assign
  buildings to upgrades per year and write per-year metadata + load symlinks.
- utils/pre/generate_adoption_scenario_yamls.py: CLI stub that will
  generate year-indexed scenario YAML entries from a base scenario.
- rate_design/hp_rates/ny/config/adoption/nyca_electrification.yaml:
  NYISO Gold Book 2025 adoption trajectory config (placeholder fractions).
- tests/pre/test_materialize_mixed_upgrade.py: skipped test stubs covering
  assignment logic, metadata combination, symlink creation, and validation.
- Justfile adoption trajectory recipes: materialize-adoption,
  generate-adoption-scenarios, run-adoption-scenario, run-adoption-all.
…option

Assigns baseline buildings to upgrades using a monotonic random-seed
allocation (contiguous shuffled-rank bands, one per upgrade), then writes
per-year materialized data that run_scenario.py can consume unchanged:

- metadata-sb.parquet: rows from each upgrade's metadata filtered to
  the buildings assigned to it for that year, combined into one file.
- loads/: symlinks pointing each building to the correct upgrade's
  load parquet ({bldg_id}-{N}.parquet), discovered from the source
  directory rather than hard-coding a filename format.

Buildings with postprocess_group.has_hp == True in the upgrade-0
baseline are excluded from random assignment and pinned to upgrade 0
in every year (they already have a heat pump and should not be
re-assigned to a new HP upgrade).

Also writes scenario_assignments.csv for reference (bldg_id +
one column per run year showing the assigned upgrade).

Validates the adoption config with buildstock_fetch.scenarios.validate_scenario
before any I/O; errors clearly if required upgrade directories are missing.
…o YAML

Reads a base scenario YAML and an adoption config, then emits a new
scenarios_<utility>_adoption.yaml with one entry per (year x run)
combination. For each generated entry:

- path_resstock_metadata and path_resstock_loads are rewritten to
  point at the materialized data produced by materialize_mixed_upgrade.
- year_run is updated to the calendar year for that adoption cohort.
- All path strings containing year=<old> are updated to year=<new>
  so MC paths resolve to the correct Cambium year.
- run_name gets a _y<year>_mixed tag inserted before the double-
  underscore tariff suffix.

Run keys follow the scheme (year_index + 1) * 100 + run_num so keys
are unique across (year, run) combinations and predictable when passed
to run-adoption-scenario.
@sherryzuo sherryzuo force-pushed the sz/mixed-upgrade-adoption-trajectory branch from fbd2b88 to 82b23c0 Compare March 22, 2026 20:20
…logistic fit

Digitizes NYISO Gold Book 2025 NYCA electrification forecast, fits a
logistic S-curve per technology, and writes the adoption config YAML
with fractions evaluated at the Cambium 5-year run years (2025–2050).
2025 is forced to 0.0 (all buildings at upgrade-0 baseline).

Also saves a curve-fit diagnostic plot alongside the YAML, and wires
a `fit-adoption-config` recipe into the shared hp_rates Justfile so
the YAML is always script-generated rather than hand-written.
- Fix run-adoption-scenario: add required --state, --output-dir, and
  run log (consistent with run-scenario)
- Fix run-adoption-all: iterate over year-indexed keys
  (year_index+1)*100+run_num instead of base run nums; reads run_years
  count from adoption config via inline Python
- Update generate-adoption-scenarios and run-adoption-all defaults to
  runs="1,2,5,6"
- fit-adoption-config: wire --stacked-plot-output arg
- data/resstock: add convenience shortcuts and end-to-end recipe for
  adoption-trajectory upgrades 01, 04, 05 (NY)
40 tests covering building assignment fractions/monotonicity, metadata
combination, symlink creation, scenario CSV output, validation error
paths, config parsing, and load file map scanning.
Add `residual_cost_frac: float | None` to `ScenarioSettings`. When set,
skip `utility_revenue_requirement` parsing (using a sentinel `rr_total=0.0`)
and pass `residual_cost_frac` through to `_return_revenue_requirement_target`;
derive `revenue_requirement` from `costs_by_type["Total System Costs ($)"]`
when the call returns `None` (i.e. 0% residual = RR equals total MC).

Enforce mutual exclusion: raise `ValueError` if both `residual_cost_frac`
and a non-empty `utility_revenue_requirement` are present in the YAML run.
Empty/none/null values for `utility_revenue_requirement` are treated as absent.

Existing runs are unaffected (`residual_cost_frac` defaults to `None`).
For each non-baseline upgrade in the mixed-upgrade materialization,
build the candidate pool from buildings where has_hp=True in that
upgrade's metadata. This prevents GSHP being assigned to ductless
buildings and dual-fuel ASHP being assigned to electrically-heated
buildings, both of which carry only baseline loads in their upgrade=N
data. Monotonicity is preserved: the per-upgrade pool is shuffled once
and adoption expands by taking the first N elements each year.

Also update the data/resstock Justfile to reflect that approximation
is skipped for upgrades 4 and 5 (only upgrade 01 is approximated).
Tests cover:
- _replace_year_in_value: both year= (Hive) and t= (Cambium) token replacement
- --residual-cost-frac: sets field and clears utility_revenue_requirement
- --cambium-supply: rewrites supply MC paths for supply runs, clears bulk TX
- --cambium-dist-mc-base: constructs Cambium dist MC path per utility/year
- End-to-end t= token replacement across adoption years
- data/resstock/Justfile: remove trailing whitespace
- nyca_electrification.yaml: yamlfmt blank line cleanup
- run_scenario.py: ruff line-length reformat
- tests/pre/__init__.py: remove spurious blank line
- fit_adoption_config.py: ruff line-length reformat
- utils/buildstock: SbMixedUpgradeScenario for CAIRO-facing materialization,
  re-exports buildstock-fetch scenario helpers
- list_adoption_years: print run years from adoption config YAML (respects
  run_years, snaps to year_labels); used by Justfile for adoption loop
- Delegate scenario materialization to SbMixedUpgradeScenario
- Add --release for release dir under path-resstock-release
- Simplify CLI and config parsing; trim docstring
- patches: use ParquetFile for timestamp read; omit schema for mixed-upgrade
  symlinked parquets (dictionary vs plain encodings vary)
- generate_utility_tx_dx_mc: fix leap-year 8760 rule (use calendar.isleap,
  not data presence); align timestamp dtype for join
- Add test_cambium_dist_mc: busbar_load rename, empty bulk TX path
- generate_adoption_scenario_yamls: add --cambium-supply, --cambium-dist-mc-base,
  --adoption-tariff-dir, --residual-cost-frac; Cambium GEA/BA options
- create_scenario_yamls: support adoption tariff dir
- Extend tests for generate/create and scenario config
- path_resstock_root, resstock_release_key; materialize-adoption uses root+release
- generate-adoption-scenarios: Cambium supply/dist, 0% residual, adoption tariff dir
- run-adoption-all: precalc tariffs per year, seasonal derivation, Cambium dist MCs
- Add scenarios_nyseg_adoption.yaml for NYSEG adoption runs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add mixed-upgrade HP adoption trajectory to rate design pipeline

1 participant